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(57) Abstract: A system and method of locating the display area of a user interactive, 
computer controlled display system including a computing device (10) for generating 
image data (lOA) corresponding to a plurality of selected images, a display area (lOB) 
for displaying the plurality of selected images (20), an image capture device (13) for 
capturing a plurality of images (21) within its capture area each captured image includ- 
ing one of the selected images and a display area locator (14) for locating the display 
area within the capture area by deriving constructive and destructive feedback data (22) 
from image data corresponding to the plurality of captured images. 
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METHOD FOR ROBUST DETERMINATION OF VISIBLE POmTS OF A 
CONTROLLABLE DISPLAY WITHIN A CAMERA VIEW 

5 

FIELD OF THE INVENTION 

The present invention relates to a computer controllable display system and in particular 
to the interaction of a user with a computer controlled image display or projected image. 

10 BACKGROUND OF THE INVENTION 

Computer controlled projection systems generally include a computer system for 
generating image data and a projector for projecting the image data onto a projection screen. 
Typically, the computer controlled projection system is used to allow a presenter to project 
presentations that were created with the computer system onto a larger screen so that more than 
15 one viewer can easily see the presentation. Often, the presenter interacts with the projected 

image by pointing to notable areas on the projected image with their finger, laser pointer, or some 
other pointing. 

The problem with this type of system is that if a user wants to cause any change to the 
projected image they must interact with the computer system using an input device such as a 

20 mouse, keyboard or remote device. For instance, a device is often employed by a presenter to 
remotely control the computer system via infrared signals to display the next slide in a 
presentation. However, this can be distracting to the viewers of the presentation since the 
presenter is no longer interacting with them and the projected presentation and, instead, is 
interacting with the computer system. Often, this interaction can lead to significant interraptions 

25 in the presentation. 

Hence, a variation of the above system developed to overcome the computer-only 
interaction problem allows the presenter to directly interact witli the projected image. In this 
system, the computer system generates image data (e.g. presentation slides) to be projected onto a 
screen (e.g. projection screen) with an image projector device. The system also includes a digital 
30 image capture device such as a digital camera for capturing the projected image. The captured 
projected image data is transmitted back to the computing system and is used to determine the 
location of the an object in front of the screen such as a pointing device. The computer system 
may then be controlled dependent on die determined location of the pointing device. For 
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instance, in U.S. Psd&Wmo. 5,138,304 assigned to the assignee of th^lbject application, a light 
beam is projected onto the screen and is detected by a camera. To determine the position of the 
light beam, the captured image data of the projected image and the original image data are 
compared to determine the pointer position. The computer is then caused to position a cursor in 
5 the video image at the pointer position or is caused to modify the projected image data in 
response to the pointer position. 

In order to implement a user interactive, computer controlled display or projection 
system, an initial calibration step is necessary where the computer system determines the location 
of the screen or more specifically the area in which the image is displayed within the capture area 
10 of the camera. By determining the location of the display area within the capture area, the system 
can then identify any object that is not part of the displayed image (e.g., objects in front of the 
displayed image). In one prior art systein, the location of the display area within the capture area 
is determined by using complex, and sometimes fragile, pattern matching algorithms requiring a 
significant amount of computing time and system memory usage. 

15 The present invention is an efficient, robust, and simple technique for locating an image 

display area within the capture area of an image capture device in a user interactive, computer 
controlled display or projection system. 

SUMMARY OF THE PiVENTION 

A system for locating an user interactive, computer controlled image display area includes 
20 a display area for displaying a plurality of selected digital images, a computing system for 

providing digital image data for causing the plurality of selected digital images to be displayed, 
an image capture device having an image capture area for capturing a plurality of images within 
the capture area each including one of the selected digital images, and a display area locator 
including a means for deriving constructive or destructive feedback data from digital image data 
25 corresponding to the plurality of captured images to determine the location of the display area 
within the image capture area. 

The method for determining the location of the user interactive, computer controlled 
display area within an image capture area of an image capture device is performed by displaying 
a plurality of selected images in the display area, capturing a plurality of images within the 
30 capture area each including one of the plurality of selected images, and deriving constructive or 
destructive image feedback data from the image data corresponding to the pluraUty of captured 
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images in order to loclRhe display screen in the image capture are^^e image capture 
device. 

BRIEF DESCRIPTION OF THE DRAWINGS 

5 Fig. 1 illustrates a block diagram of a first embodiment of a system for locating a display 

area in a user interactive, computer controlled display system; 

Fig. 2 illustrates a first embodiment of the method of locating a display area within a 
capture area in a user interactive, computer controlled display system in accordance with the 
present invention; 

10 Fig. 3 iUustrates a second embodiment of the method of locating a display area within a 

capture area in a user interactive, computer controlled display system in accordance with the 
present invention; 

Figs. 4A and 4B illustrate a third embodiment of the method of locating a display area 
within a capture area in a user interactive, computer controlled display system in accordance with 
15 the present invention; 

Fig. 5A shows an exemplary image of a display area in a capture area; 

Fig. SB shows an image corresponding to image data obtained by performing the method 
of the present invention; and 

Figs. 6A and 6B illustrate selected images for displaying in a fourth embodiment of the 
20 method of locating a display area within a capture area in a user interactive, computer controlled 
display system in accordance with the present invention. 
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tJWItl.ed description of the inveM[qn 



Figure 1 shows a block diagram of an interactive computer controlled image display 
system which includes a computing system 10 for generating image data lOA and a graphical 

5 interface 11 for causing selected images lOB corresponding to the image data lOA to be 
displayed in display area 12. It should be understood that the graphical interface may be a 
portion of the computing system or may be a distinct element external to the computing system. 
The system further includes an image capture device 13 having an associated image capture area 
13A for capturing a plurality of images each including one of the plurality of selected displayed 

10 images lOB. The captured images also include images IOC that are not within the display area. 
Non-display area images include anything other than what is being displayed within the display 
area. The plurality of captured images is converted into digital image data 13B which is 
transmitted to a display area locator 14. Display locator 14 includes a means for deriving 
constmctive and destructive image feedback data 15 from the captured image data 13B to 

15 determine the location of the display area 12 within the image capture area 13A. The display 

locator can then optionally transmit the screen location information 14A to the computing system 
10 for use in the user interactive, computer controlled display system. 

In this embodiment, the computing system includes at least a central processing unit 
(CPU) and a memory for storing digital data (e.g., image data) and has the capability of 
20 generating at least three levels of grayscale images. The display area can be a computer monitor 
driven by the graphical interface or can be an area on a projection screen or projection area (e.g., 
a wall). In the case in which images are displayed using projection, the system includes an image 
projector (not shown in Fig. 1) that is responsive to image data provided from the graphical 
interface. 

25 In one embodiment, the image capture device is a digital camera arranged so as to capture 

at least all of the images lOB displayed in the display area 12 within a known time delay. It is 
well known in the field of digital image capture that an image is captured by a digital camera 
using an array of sensors that detect the intensity of the light impinging on the sensors within the 
capture area of the camera. The light intensity signals are then converted into digital image data 

30 corresponding to the captured image. Hence, the captured image data 13B is digital image data 
corresponding to the captured image. In another embodiment the image capture device can be a 
digital video camera capable of capturing still unages of the display area. 
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In one embodHRit, the selected images lOB are a plurality^Ringle intensity grayscale 
images that are successively displayed in the display area and captured by the image captaire 
device. 

It should be understood that all or a portion of the functions of the display locator 14 can 
5 be performed by the computing system. Consequently, although it is shown external to the 
computing system, all or portions of the display locator 14 may be implemented within the 
computing system. 

It should be farther understood that the display locator can be implemented in a software 
implementation, hardware implementation, or any combination of software and hardware 
10 implementations. 

Figure 2 shows a first embodiment of a method of determining the location of the display 
area within the capture area of an image capture device in an user interactive, computer 
controlled display system. According to this embodiment, a plurality of selected images is 
displayed within a display area (block 20). A pluraUty of images is captured within the capture 
15 area of an image capture device each captured image including one of the selected images (block 
21). Constructive and destmctive feedback data is derived from image data corresponding to the 
captured images to locate the display area within the capture area (block 22). 

Fig. 3 shows a second embodiment of a method of determining the location of the display 
area within the capture area of an image capture device in an user interactive, computer 

20 controlled display system. In tliis method, at least three single intensity grayscale images are 
displayed within the display area (block 30). A plurality of images are captured within the 
capture area of the image capture device each including one of the, at least three, single intensity 
grayscale images (block 31). Constmctive or destructive feedback data is derived by storing 
image data corresponding to a first captured image including a first one of the, at least three, 

25 images in a pixel array (block 32) and by incrementing or decrementing the pixel values 

dependent on the image data corresponding to the remainder of the captured images including at 
least the second and third single intensity grayscale images (block 33). As shown, block 33 
generates image data showing the location of the display area within the capture area. It should 
be understood that constructiv^e feedback infers that a given pixel value is incremented and 

30 destmctive feedback infers that a given pixel value is decremented. In one embodiment, pixel 
values within the array that correspond to the display area are incremented by a first 
predetermined constant value and pixel values within the array that correspond to the non-display 
area are decremented by a second predetennined constant value. In one variation of the third 
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embodiment, the fee JHR: can be applied iteratively. This iterative^Bcess is achieved by 
redisplaying at least second and third images and again incrementing or decrementing pixel 
values in the array. 

Figure 4A shows a third embodiment of a method of the present invention and Fig, 4B 
5 illustrates corresponding operations perfomied by the method. Initially, an array (x, y ) of 

digital storage is initialized within a digital memory storage area wherein each element within the 
array stores a value corresponding to each pixel location within the image capture area of the 
capture device. The initialization of the array sets all values within the array to zero such that 
AXjc,y)=0 (block 40A and 40B). 

10 A first single intensity grayscale image having an intensity of (where 

^HicH > ^mid > how ) is displayed in the display area 12 and a first image is captured within image 
capture area 13 A having a corresponding array of image data A^^ (x% y) (block 41 A and 41B). 
The first captured image includes a first single grayscale image displayed in display 12. 
Specifically, as shown by 41B the first captured image includes an area of pixels s that 

15 corresponds to the display area 12 all having an intensity of and an area of pixels s that 
corresponds to the non-display area. Array A^^ (x, y ) is also stored. 

Next, a second single intensity grayscale image is displayed in display area 12 having a 
maximum intensity level of /^^^ff a second image is captured within the image capture area 
having a corresponding image data array A^^i^, y) (block 42A). The array A^^i^, y) includes 
20 pixels s having intensities of 7;^;^// non-display area pixels s (block 42B). The pixel values 
of array (x, y) are compared with corresponding pixel values of A^^(x, y) (block 43) and the 
values of A^ (x, y) are incremented or decremented (block 44A) according to the following 
conditions: 

if A,, (x, y) > A,i (x, y ), then A, (x, y )= A, (x, y )+ c^, ; condition 1 

25 otherwise A^ (jc, y )= A^ (x, y )+ c„,^ ; condition 2 

where c„^^ is a negative integer and c^^^ is a positive integer. In one embodiment, c^^, = 20 and 
<^nes = . Restated, A, (x, y )= A^ (x, y )+ c j^^^^^^ . 
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It should be nowRhat the value of (x, y) is dependent on t^^ynamic behavior of 
corresponding pixel values from the first captured image to the third captured image in the 
display and non-display areas. Consequently, since the intensity level of the display area pixels 
s in the second captured image (42B) is If^j^f^ and since the intensity level of the display area 
5 pixels s in the first captured image (41B) is I^., , then A^^ (x, y) > A^ (x, y) in the display area s . 
As a result, condition 1 applies and c^^, is added to all values corresponding to the display area 
in array A^ (jc, y). Since (x, y) is initialized such that all values are set.to 0 prior to block 44A, 
then the values corresponding to s in array A^(x, y) are incremented to a value of c^^^ after the 
first feedback data is applied. 

10 Alternatively, assuming the background of the display area does not change, the intensity 

of the non-display area pixels ~s will be the same and A^^{x,y)=A^^ix,y). Consequently, 
condition 2 applies and c,,^^ is added to the values corresponding to non-display pixels in array 

(x, y). However, in some cases, condition 2 may apply due to image noise or if the 
background changes. Consequently, values of A^ corresponding to non-display area s will 

15 range in value from (c^, {f „,^ ). 

Hence, constructive feedback is appHed to all values corresponding to the display area in 
array A^ (x, y) and, for the most part, destructive feedback is applied to values corresponding to 
non-display ai-ea in array A^ (x, y). As a result, display area values are driven positively and non- 
display area values are driven in negatively. 

20 Next, a third single intensity grayscale image having an intensity level of I^q^ is 

displayed in display area 12 and a third image is captured in capture area 13A having a 
corresponding array of image data A^j (x, y) (block 45 A and 45B). As shown the pixel array 

A^3(x,y) includes pixels s having intensities of /^^ and pixels s (45B). The pixel values of 
array A^i(x, y) are then compared with corresponding pixel values of A^^{x, y) (block 46) and 
25 the values of A^ (x, y ) are incremented or decremented (block 47 A) according to the following 
conditions: 

if A,3(x,y) < A^^{x,y) then A,(x,y)= AX^%y)+Cp^, ; condition 3 

otherwise A^ (x, y )= A^ (x, y )+ c^^ . condition 4 

1 - 
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Restated, A, (x, y)= A^y)+ Cj^^^ . Note conditions 3 and 4 are^^Kntially the inverse of 
conditions 1 and 2. As a result, since the intensity of the display area pixels s in the third 
captured image data (45B) is /tow and since the intensity of the display area pixels s in A^, (x, y) 
is/,„,^ , then A^3(jc, y) < A,i (x, y) in display area^ . As a result, condition 3 applies and is 
5 once again added to all values corresponding to the display area s in array AXx,y). Hence, 
constructive feedback is applied to the display area values in array A, (x, y) when applying both 
of the first and second feedback data. 

In contrast, assuming again that the non-display pixel values in the third captured image 
data 45B still have not changed then, in most cases condition 4 applies and c„,g is once again 
10 added to non-display values in A„ (x, y). In the case in which the non-display area image has 
changed or due to noise, condition 3 may apply. However, destructive feedback, for the most 
part, is applied to the non-display area values in array A, (x, y) when applying both of the first 
and second feedback data. 

The net effect of feedback blocks 44A and 47A is that the values in A^ (x, y) 
15 corresponding to the display area s are driven positively while the values of the non-display area 
s are driven negatively. Fig. 4A also shows that an iterative loop 48 can be performed such that 
for each of the second and third captured images additional feedback is added to A, (x, y) such 
that if / is the number of iterations in the loop 48, then: 



A^i.x,y) = \ , s 
i oo Vfyx, y)^ s, -< 



20 The resulting array A^^ (x, y) will comprise very positive values in pixel locations 

corresponding to the display area and very negative values in pixel locations corresponding to 
non-display areas. Hence, a threshold comparison can be used to identify extremely 
positive/negative values and hence can be used to identify the location of the display area within 
the capture area of the image capture device. 

25 Fig. 5A illustrates an exemplary image of a display area 12 captured within a capture area 

13A. In particular, the display aiea 12 is a computer monitor. Fig. 5B represents the data (i.e., 
Ajirua C^' y)) providing the location information of the display area 12. As can be seen, the 
display area 12 is white corresponding to pixel values having a maxrmum intensity level of /htgh 

8 
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and the non-display arSHIrrounding the display area within the capttWarea 13A is black 
corresponding to pixel values having a minimum intensity level of . 

In a fourth embodiment of the method of the present invention, the second and third 
images are not displayed with a single grayscale intensity. Instead, the second image is displayed 
5 such that a first region of the image is displayed to have a first intensity value and a second 

region is displayed to have second intensity value while the third image is displayed to have the 
converse intensities in corresponding regions. For instance, Figs. 6A and 6B show display area 
12 within the capture area 13A, and the manner in which the second and third images are 
displayed for the fourth embodiment of the method of the present invention. As shown in Fig. 

10 6A, the second image 60 is displayed within display area 12 such that a region of the display 
area 12 is set to a grayscale intensity value corresponding to /^^^^ (i.e., white) and the other 
region is set to a grayscale value corresponding to Z^^^iv (i-®-' black). It should be noted that 
region P^ essentially frames the display area 12. Region P^ corresponds to the non-display area. 
Conversely, the third image 61 shown in Fig. 6B is displayed such that region P^ is a set to a 

15 grayscale intensity value corresponding to I^y^ and region P^ is set to a grayscale intensity 
value corresponding to ImaH • 

The effect of displaying the second and third images as shown in Figs. 6A and 6B is to 
obtain final array data A^„^; (jc, y) that identifies the location of a display area in the capture area 
as the display area region P^ with a non-fuzzy or "clean" border ai'ound it. In particulai', by using 
20 the method as shown in Figs. 4A and 4B and displaying the second and third images as shown in 
Figs. 6A and 6B, the display area P^ is driven positively, the non-display screen area P^ is driven 
negatively, and the display area border P^ is driven even more negatively than the non-display 
screen area such that a sharper border is established about display area P^ . 

The second, third and fourth embodiments can be implemented such that array A^(x, y) is 
25 normalized prior to iteratively applying the first and second feedback (blocks 43 A-48A). 

Normalization is performed by subtracting a static constant value from all pixel values in the 
array as follows: 

9 
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This nonnalization is^lfformed so as to better discriminate betweeiW|ions that have 
constructive feedback and those that do not. In other words, normalization is performed to make 
the iterative feedback more effective in providing optimized final array data for identifying 
display and non-display areas. 

5 In another embodiment, the system and method can be implemented such that instead of 

applying constructive feedback to display area values and destructive feedback to non-display 
area values in array (x, y), the inverse feedback conditions apply. Specifically, destructive 
feedback is applied to the display area values while constructive feedback is applied to non- 
display area values in order to identify the non-display area within the capture area of the image 

10 capture device. 

In still another embodiment, the system and method can be implemented such that instead 
of comparing the pixel values of the stored first image to corresponding pixel values of the 
second and third captured image data (Figs. 4A and 4B, blocks 43 and 46), first captured image 
pixel values are compared to noise adjusted second and third captured image pixel values, where 
15 noise adjusted pixel values are the original pixel value plus a noise factor value, . For 
instance, conditions 1 and 3 become the following: 

if {x, y)+ > A,i (x, y), then A, {x, y)= A, {x, y)+ c^,, ; condition 1 ' 

if (x, y)+ c„f < A,j {x, y) then A, {x, y)= A, {x, y)+ c^^ ; condition 3 ' 

where c^ is a constant value that takes into account minor fluctuations in pixel values within the 
20 user interactive, computer controlled display system due to lighting conditions, camera quality, and 

the quality and type of displayed image. The addition of the noise factor results in a more robust 

implementation of the system and method of the present invention. 

Hence, a system and method is described that provides an arithmetically non-complex 

solution to finding the display area within the capture area of an image capture device in a user 
25 interactive, computer controlled display system. 

In the preceding description, numerous specific details are set forth, such as feedback 

constants, intensity values, and image types in order to provide a through understanding of the 

present invention. It wiU be apparent, however, to one skilled in the art that these specific details 

need not be employed to practice the present invention. la other instances, well-known image 
30 processing techniques have not been described in detail in order to avoid unnecessarily obscuring 

the present invention. 

10 
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In addition, araRgh elements of the present invention hav^RIn described in 
conjuncdon with certain embodiments, it is appreciated that the invention can be implement in a 
variety of other ways. Consequently, it is to be understood that the particular embodiments 
shown and described by way of illustration is in no way intended to be considered limiting. 
5 Reference to the details of these embodiments is not intended to limit the scope of the claims 
which themselves recited only those features regarded as essential to the invention. 
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CLAIMS 

I claim: 

1. A method for determining the location of a computer controlled display 
area within an image capture area of an image capture device: 

Displaying (20) a pluraUty of selected images within the display area; 

capturing a plurality of images (21) within the capture area of the 
image capture device each including one of the displayed selected 
images; and 

deriving constructive and destructive feedback data (22) from image 
data corresponding to the plurality of captured images. 

2. The method as described in Claim 1 wherein displaying a plurality of 
selected images comprises displaying a plurality of selected images each 
having an associated single intensity grayscale value. 

3. The method as described in Claim 2 wherein displaying the plurahty of 
selected images (30) comprises displaying a first selected image having a 
first single intensity grayscale value Ii, displaying a second selected image 
having a second single intensity grayscale value I2, and displaying a third 
selected image having a third single intensity grayscale value I3, wherein h 
> Ii > I3. 

4. The method as described in Claim 3 wherein the second single intensity 
grayscale value corresponds to one of a maximum intensity level and a 
minimum intensity level, the third single intensity grayscale value 
corresponds to the other of the maximum intensity level and the minimum 
intensity level, and the first single intensity grayscale value is a value 
between the maximum and minimum intensity values. 

5. The method as described in Claim 3 wherein displaying a plurality of 
selected images comprises displaying each of the second selected and the 

12 
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Attorney Docket No^^07846 

third selected images such that first regions have a corresponding intensity 
value h and second regions have a corresponding intensity value I3. 

6. The method as described in Claim 3 wherein deriving constructive and 
destructive feedback data from the image data comprises: 

storing the image data (32) corresponding to a first captured image of 
the plurality of captured images which includes the first selected image 
in a pixel array associated with the capture area; and 

one of incrementing and decrementing pixel values (33) in the pixel 
array dependent on the image data corresponding to a second captured 
image of the plurality of captured images including the second selected 
image and dependent on the image data corresponding to a third 
captured image of the plurality of captured images including the third 
selected image. 

7. The method as described in Claim 6 wherein the pixel values are 
incremented by a first predetermined constant value and the pixel values 
are decremented by a second predetermined constant value. 

8. The method as described in Claim 6 wherein pixel values corresponding to 
the display area within the capture area are incremented so as to create the 
constructive feedback data and pixel values not corresponding to the 
display area are decremented so as to create the destmctive feedback data. 

9. The method as described in Claim 6 wherein incrementing and 
decrementing pixel values further comprises comparing captured image 
data of the first captured image to image data corresponding to each of the 
second captured image and the third captured image and incrementing and 
decrementing dependent on the comparison results. 

10. The method as described in Claim 9 wherein comparing further comprises 
comparing pixel values of the first captured image data to corresponding 
noise adjusted pixel values of each of the second captured image and the 
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third captured image data, wherein the noise adjusted pixel value 
comprises the original pixel value plus a noise factor value. 

11. The method as described in Claim 6 wherein the display area is a 
projection screen and the image capture device is a digital camera. 



BNSDOCID: <WO ^0206538aA2_L> 



14 



wo 02/065388 



PCT/US02/02716 



1/6 



IOC 



COMPUTING 
SYSTEM 
10 



GRAPfflCS 
INTERFACE 
11 



lOB 



14A 



lOA 



12' 



13A' 



13B 



DISPLAY LOCATOR 
14 



FEEDBACK DATA 
DERIVATOR 15 



IMAGE CAPTURE 
DEVICE 
13 



FIG. 1 



DISPLAY A PLURALITY OF 
SELECTED IMAGES IN 
DISPLAY AREA 
20 



CAPTURE A PLURALITY OF 
IMAGES EACH INCLUDING 
ONE OF THE SELECTED 
IMAGES WirHIN THE 
CAPTURE AREA 
21 



DERIVE CONSTRUCTIVE AND 
DESTRUCTIVE FEEDBACK 
DATA FROM IMAGE DATA 
CORRESPONDING TO THE 
CAPTURED IMAGES 
22 



FIG. 2 



<wo. 



.0206538aA2J_> 



wo 02/065388 



PCT/US02/02716 



2/6 



DISPLAY AT LEAST THREE 

SINGLE INTENSITY 
GRAYSCALE IMAGES IN THE 
DISPLAY AREA 
30 



CAPTURE A PLURALITY OF 
IMAGES EACH INCLUDING 
ONE OF THE AT LEAST THREE 

IMAGES wrrmN the 

CAPTURE AREA 
31 



STORE IMAGE DATA 
CORRESPONDING TO A FIRST " 

CAPTURED IMAGE 
INCLUDING A FIRST ONE OF 

THE AT LEAST THREE 
IMAGES IN A PIXEL ARRAY 
32 



INCREMENT OR DECREMENT 
PIXEL VALUES IN THE 
ARRAY DEPENDENT ON 
IMAGE DATA 
CORRESPONDING TO 
CAPTURED IMAGES 
INCLUDING AT LEAST THE 
SECOND AND THIRD 
GRAYSCALE IMAGES 
• 33 



IMAGE DATA SHOWING 
LOCATION OF DISPLAY AREA 
IN THE CAPTURE AREA 



FIG. 3 



BNSDOCID: <WO ^0206538aA2J_> 



wo 02/065388 



MTTIALIZE ARRAY 
40A 



DISPLAY AND CAPTURE 

A FIRST SINGLE 
INTENSITY GRAYSCALE 

IMAGE HAVING I=Imid 

41A 



DISPLAY AND CAPTURE 
A SECOND SINGLE 
ESTTENSITY GRAYSCADE 

IMAGE HAVING I=IfflGH 

42A 



COMPARE 
CORRESPONI 

OFAc2(x,y)T 


" "1 

)ING PIXELS 
O Aci(x,y) 

43 




r 



INCREMENT AND 
DECREMENT VALUES OF 
Ac(x,y) DEPENDENT ON 
COMiPARISON 

44A 



CONT. IN 
nG.4B 



i 



3/6 



PCT/US02/02716 

(40B) 

Ac(x,y)= 0 



(41B) 



FIRST 
-►CAPTURED 
IMAGE DATA 
Aci(x,y) 





1 


— i 

s 


^ 13A 


S : I=ImiD 





12 



(42B) 



SECOND 
-►CAPTURED 

IMAGE DATA 

Ac2(x,y) 





1 
1 
t 


\s 








s 


13A 


S : I=IhIGH 





12 



ITERATIVE LOOP 
48 



(44B) 

-► Ac(x,y)=Ac(x,y)+CFEEDBACKl 



A CONT. FROM 
! FIG.4B 



FIG. 4A 



BNSDCXJID: <WO 02065388A2J_> 



wo 02/065388 



PCT/US02/02716 



4/6 



CONT. FROM 
FIG.4B 



DISPLAY AND CAPTURE 
A THIRD SINGLE 
INTENSITY GRAYSCALE 

IMAGE HAVING I=IloW 

45A 



COMPARE 




CORRESPONDING PIXELS 


OFAc3(x,y)TO Aci(x,y) 




46 




r 



INCREMENT OR 
DECREMENT VALUES OF 
Ac(x,y) DEPENDENT ON 
COMPARISON 

47A 



Acfinal(X,y) 



CONT. IN FIG. 4A 



THIRD 
> CAPTURED 
IMAGE DATA 

Ac3(x,y) 



(45B) 




S : I=IlOW 



ITERATIVE LOOP 
48 



(47B) 

^ Ac(X,y)=Ac(x,y)+CFEEDBACK2 



FIG. 4B 



BNSDOCID: <WO 02065388A2_L> 



wo 02/065388 ^ PCT/US02/02716 



5/6 




FIG. 5A 




FIG. 5B 



BNSCXDCID: <WO ^0206538aA2_L> 





FIG. 6B 



BNSDCXDID: <WO 0206538aA2_L> 



(12)1NTERNAT10N 



PLICATION PUBLISHED UNDER THE PATEN 



PERATION TREATY (PCT) 



(19) World Intellectual Property Organ tzation 
International Bureau 

(43) International Publication Date 
22 August 2002 (22.08.2002) 




PCT 



llllllillllllil^ 

(10) International Publication Number 

wo 02/065388 A3 



(51) International Patent Classification^: G06K 1 1/08 

(21) International Application Number: PCT/US02/02716 

(22) International Filing Date: 29 January 2002 (29.01 .2002) 

(25) Filing Language: English 

(26) Publication Language: English 
(30) Priority Data: 



09/774,452 



30 January 200 1 (30.0 1 .200 1 ) US 



(71) Applicant: HEWLETT-PACKARD COMPANY 

[US/US]; 3000 Hanover Street, Palo Alto, CA 94303-1881 
(US). 

(72) Inventor: LIN, I^ong; 325 King's Mountain Boulevard, 
Woodside, CA 94062 (US). 



(74) Agent: HEMINGER, Susan. E.; Hewlett-Packard Com- 
pany, Legal Department, IP Section, P.O. Box 272499, Ft. 
Collins, CO 80527-2400 (US). 

(81) Designated States (national): AE, AG, AL, AM, AT. AU, 

AZ. BA, BB, BG, BR, BY, BZ, CA, CH, CN, CO, CR, CU, 
CZ, DE, DK, DM, DZ, EC, EE, ES, H, GB, GD, GE, GH, 
GM, HR, HU, ID. IL, IN, IS, JP, KE, KG, KP, KR, KZ, LC, 
LK, LR, LS, LT, LU, LV, MA, MD, MG, MK, MN, MW, 
MX, MZ, NO, NZ, OM, PH, PL, PT, RO, RU, SD, SE, SG. 
SI, SK, SL, TJ. TM, TN, TR. XT, TZ, UA, UG, UZ, VN, 
YU, ZA, ZM, ZW. 

(84) Designated States (regional)'. ARIPO patent (GH, GM, 
KE. LS, MW, MZ, SD, SL, SZ, TZ, UG, ZM, ZW), 
Eurasian patent (AM, AZ, BY, KG, KZ, MD, RU. TJ, TM), 
European patent (AT, BE, CH, CY, DE, DK, ES. FI, FR, 
GB, GR, IE, IT, LU, MC, NL, PT, SE, TR), OAPI patent 
(BF, BJ. CF, CG, CI, CM. GA, GN, GQ, GW, ML, MR, 
NE, SN. TD, TG). 

[Continued on next page] 



(54) Title: METHOD FOR ROBUST DETERMINATION OF VISIBLE POINTS OF A CONTROLLABLE DISPLAY WITHIN 
A CAMERA VIEW 



< 

QO 
OC 

m 

IT) 

o 



DISPLAY A PLURAUTY OF 
SELECTED IMAGES IN 
DISPLAY AREA 
20 



CAPTURE A PLURALITY OF 
IMAGES EACH INCLUDING 
ONE OF THE SELECTED 
IMAGES WITHIN THE 
CAPTURE AREA 
21 



I 



DERIVE CONSTRUCTTIVE AND 
DESTRUCTIVE FEEDBACK 
DATA FROM IMAGE DATA 
CORRESPONDING TO THE 
CAPTURED IMAGES 
22 



(57) Abstract: A system and method of locating the 
display area of a user interactive, computer controlled 
display system including a computing device (10) for 
generating image data (lOA) corresponding to a plu- 
rality of selected images, a display area (lOB ) for dis- 
playing the plurality of selected images (20), an image 
capture device (13) for capturing a plurality of images 

(21) within its capture area each captured image includ- 
ing one of the selected images and a display area locator 
( 14) for locating the display area within the capture area 
by deriving constructive and destructive feedback data 

(22) firom image data corresponding to the plurality of 
captured images. 



BNSDOCID: <WO ^0206538aA3J_> 



<ft wo 02/065388 A3 llillllftlllllililiiliilllii^^ 

Published: For two-letter codes and other abbreviations, refer to the "Guid- 

— with international search report ance Notes on Codes and Abbreviations " appearing at the begin- 

— before the expiration of tire time limit for amending the ning of each regular issue of the PCT Gazette, 
claims and to be republished in the event of receipt of 

amendments 

(88) Date of publication of the international search report: 

31 October 2002 



BNSDOCID: <WO ^0206538aA3J_> 



INTERNATIONAL SEARCH REPORT 



A. CLASSIFICATION OF SUS^B^ 

PC 7 G06Kll/0^^ 



rJ^^tional Application No 

~^/US 02/02716 



ATTER 



m 



According \o International Patent Classification (IPC) or to both national classification and IPC 



B. FIELDS SEARCHED 



Minimum documentation searched (classification system followed by classification synnbote) 

IPC 7 G06K G06F 



Documentation searched other than minimum documentation to the extent that such documents are included In the fields searched 



Electronic data base consulted during the international search (name of data base and. where practical, search terms used) 

EPO-Internal , WPI Data, PAJ, COMPENDEX 



C. DOCUMENTS CONSIDERED TO BE RELEVANT 



Category ° Citation of document, with indication, where appropriate, of the relevant passages 



EP 0 718 748 A (CANON K.K.) 

26 June 1996 (1996-06-26) 

column 6, line 1 -column 7, line 33; 

figure 5 

US 6 075 895 A (QIAO ET AL.) 

13 June 2000 (2000-06-13) 

column 5, line 19 -column 6, line 7; 

figures 1,3-5 



Relevant to claim No. 



□ 



Further documents are listed in the continuation of box C. 



10 



Patent family members are listed in annex. 



" Special categories of cited documents : 

'A' document defining the general state of the art which is not 

considered to be of particular relevance 
'E* earlier document but published on or after the international 

filing date 

■L' document which may throw doubts on priority claim(s) or 
which is cited to establish the publication date of another 
citation or other special reason (as specified) 

'O" document referring to an oral disclosure, use, exhibition or 
other means 

*P* document published prior to the international filing date but 
later than the priority date claimed 



later document published after the International filing date 
or priority date and not in conflict with the application but 
cited to understand the principle or theoiy underlying the 
invention 

document of particular relevance; the claimed invention 
cannot be considered novel or cannot be considered to 
involve an inventive step when the document is taken alone 

document of particular relevance; the claimed invention 
cannot be considered to involve an inventive step when the 
document is combined with one or more other such docu- 
ments, such combination being obvious to a person skilled 
in the art. 

' document memb»er of the same patent family 



Date of the actual completion of the international search 



22 August 2002 



Name and mailing address of the ISA 

European Patent Office, P.B. 5818 Palentlaan 2 
NL - 2280 HV Riiswiil< 
TeL (+31-70) 340-2040. Tx, 31 651 epo nl. 
Fax: (+31-70) 340-3016 



Date of mailing of the international search report 



29/08/2002 



Authorized officer 



Taylor, P 



Form PCT/ISA/210 (second sheet) (July 1992) 



BNSDOCID: <WO ^0206638aA3J_> 



INTERNATIONAL SEARCH REPORT 



Patent documei 
cited in search repol 



por^ 



Pubiication 
date 



Patent fami 
member(s) 



itfonal Application No 

T/US 02/02716 



Publication 
date 



EP 718748 



26-06-1996 



JP 3270643 B2 

JP 8179875 A 

DE 69526779 Dl 

EP 0718748 A2 

US 6323839 Bl 



02-04-2002 
12-07-1996 
27-06-2002 

26- 06-1996 

27- 11-2001 



US 6075895 A 13-06-2000 AU 8141198 A 04-01-1999 

EP 0920670 Al 09-06-1999 

JP 2000517087 T 19-12-2000 

WO 9859312 Al 30-12-1998 



Foim PCT/ISA/210 (patent famii/ annex) (July 1892) 



BNSDOCID; <WO ^02065388A3_I_> 



This Page is Inserted by IFW Indexing and Scanning 
Operations and is not part of the Official Record 



Defective images within this document are accurate representations of the original 
documents submitted by the applicant. 

Defects in the images include but are not limited to the items checked: 

□ BLACK BORDERS 

□ IMAGE CUT OFF AT TOP, BOTTOM OR SIDES 

□ FADED TEXT OR DRAWING 

□ BLURRED OR ILLEGIBLE TEXT OR DRAWING 

□ SKEWED/SLANTED IMAGES 

□ COLOR OR BLACK AND WHITE PHOTOGRAPHS 

□ GRAY SCALE DOCUMENTS 

J^rCm^S OR MARKS ON ORIGINAL DOCUMENT 
J3rEFERENCE(S) or EXHIBIT(S) SUBMITTED ARE POOR QUALITY 



□ OTHER: 



IMAGES ARE BEST AVAILABLE COPY. 
As rescanning these documents will not correct the image 
problems checked, please do not report these problems to 
the IFW Image Problem Mailbox. 



BEST AVAILABLE IMAGES 




